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This report presents an approach to evaluating networked improvement communities 
(NICs) and describes the tools used to apply the approach to a formative evaluation 
of the Minnesota Alternative Learning Center Networked Improvement Community 
(Minnesota ALC NIC). In fall 2019 leaders at the Minnesota Department of Education 
requested support from the Regional Educational Laboratory (REL) Midwest to 
address low high school graduation rates among students in alternative learning 
centers (ALCs). In partnership with five ALCs, the department and the REL Midwest 
participated in the Minnesota ALC NIC to raise graduation rates at ALCs by improving 
their use of competency-based credit recovery practices. REL Midwest researchers 
then developed an approach for evaluating networked improvement communities 
(NICs) and applied it to the Minnesota ALC NIC. This report shares the findings from 
the evaluation, using them to illustrate the process of analyzing and interpreting the 
data collected on NIC implementation, and discusses the limitations of the tools. 
The approach can be adapted to evaluate other NICs in prekindergarten—grade 12 
education. The tools require basic quantitative and qualitative analysis skills, and 
evaluators or facilitators can adapt them to provide formative feedback on whether 
an NIC is operating as intended. 
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BACKGROUND 


Interest is growing in the use of improvement science to address the complex problems that 
arise in education practice. Improvement science, which entails cycles of activities designed 
to test promising practices, was developed initially to enhance industrial manufacturing, 

but it has since been applied in fields such as medicine and engineering and, more recently, 
in education (Bryk, 2015). One common approach is to use improvement science processes 
within collaborative research partnerships, known as networked improvement communities 
(NICs), which engage in collective efforts to improve systems (Engelbart, 1992; see box 1 for 
definitions of key terms). 


Using continuous improvement research methods, NIC participants identify a common 
problem of practice and work together to test and refine solutions in different contexts. 
Through this process NICs learn what works, for whom, and under what conditions (Bryk et 
al., 2015). When the process is well implemented, NICs accelerate participants’ capacity to 
learn from one another, which can lead to improvements in practice (Bryk et al., 2015; Russell 
et al., 2017). While NICs are increasingly being used in education, little research has been con- 
ducted on their implementation or outcomes in prekindergarten (preK)—grade 12 education. 


The Minnesota Alternative Learning Center 
Networked Improvement Community 


In 2018 the Minnesota Department of Education requested support from the Regional Edu- 
cational Laboratory (REL) Midwest to identify ways to increase high school graduation rates 
among students taking courses that they had previously failed. A statewide review of programs 
that support students in completing courses—also known as credit recovery programs— 
revealed that students in need of credit recovery often moved from a traditional or charter 
high school to one of more than 200 state-approved alternative programs, also known as 
alternative learning centers (ALCs), which serve secondary students who are off track to gradu- 
ate, often because they have failed one or more high school courses. However, ALCs in Minne- 
sota had lower high school graduation rates than the state’s traditional or charter high schools. 


As a result of the review, ALC education leaders in the Minnesota Department of Education 
made improving the graduation rates of high school students in ALCs a priority. They chose 
to focus on implementation of competency-based education, which they believed held 
promise for improving outcomes among the students they served. Unlike students in tra- 
ditional preK—12 education, who attend classes at the same time as their peers and receive 
credit for a course based on participation and effort, students in competency-based courses 
have greater autonomy to determine where and when they learn and the pace of learning. 
There are differences, as well, in the types of assessments used to measure learning (Haynes 
et al., 2016); students in ALCs receive credit for a course when they demonstrate mastery of 
the competencies identified for that course. 


In response to the review, the Minnesota Department of Education and the REL Midwest, 
in partnership with five Minnesota ALCs, participated in the Minnesota Alternative Learning 
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Background 


Box 1. Definition of key terms 


Aim. A measurable outcome the network is trying to achieve. 


Alternative learning center (ALC) teams. Schools that serve secondary students who are off track to 
graduate, often because they have failed one or more high school courses. ALC teams are groups of indi- 
viduals from each ALC working together on Plan-Do-Study-Act cycles. 


Change idea. The practices tested using Plan-Do-Study-Act cycles. By testing similar change ideas in 
different contexts, networked improvement community (NIC) participants learn what works, for whom, 
and in what context. 


Continuous improvement milestone. Key activities that participants carry out as part of the continuous 
improvement research process. These milestones are described in box 2. 


Continuous improvement research. An approach to research based on improvement science principles 
and iterative cycles of testing practices (Langley et al., 2009). Plan-Do-Study-Act cycles are among many 
models of continuous improvement research (Park et al., 2013). 


Driver diagram. A visual display of NIC participants’ theory of the factors that likely contribute to achiev- 
ing their aim. Primary drivers contribute directly to achieving the aim, while secondary drivers are compo- 
nents of primary drivers. 


Fishbone diagram. Used to create a visual display of the root causes of a problem of practice. A state- 
ment of the problem is placed at the head of the fishbone. Each major “bone” represents hypothesized 
root causes of the problem. 


Improvement science. Cycles of activities designed to test promising practices for addressing challenges 
(Improvement Science Research Network, 2016). 


Minnesota Alternative Learning Center Networked Improvement Community (Minnesota ALC NIC) 
evaluation team. The Regional Educational Laboratory Midwest evaluators who developed an evaluation 
approach to assess implementation of NICs in preK-12 education and applied the approach to evaluate 
the implementation of the Minnesota ALC NIC. 


Minnesota ALC NIC facilitators. Three Regional Educational Laboratory Midwest facilitators who were 
experts in NICs and were included as participants in the Minnesota ALC NIC. The Minnesota ALC NIC facil- 
itators convened the ALC teams at the five NIC meetings and led continuous improvement activities, such 
as developing an aim statement and conducting a root cause analysis. Between NIC meetings, the Minne- 
sota ALC NIC facilitators held regular check-in meetings to support each ALC team. 


Networked improvement community (NIC). A collaborative research partnership that applies the 
principles of improvement science within networks of organizations. NIC participants engage in Plan-Do- 
Study-Act cycles to learn what works. 


Plan-Do-Study-Act cycle. Rapid cycles of designing, implementing, testing, and redesigning to learn 
about what works in different contexts, a common approach in continuous improvement research. 


Center Networked Improvement Community (Minnesota ALC NIC) to improve the use of 
competency-based credit recovery practices and thus ultimately raise high school gradua- 
tion rates for students in ALCs. Participants in the NIC included a leader from the Minnesota 
Department of Education, three facilitators from the REL Midwest, and 36 teachers and 


Background 


administrators from five ALCs in the greater Twin Cities area.' The Minnesota ALC NIC met 
four times from September 2019 to February 2020. There was full attendance at each of 
these meetings, although one ALC left the NIC after the October 2019 meeting and one ALC 
joined the NIC after the December 2019 meeting. 


A REL Midwest research team, which had developed a framework to assess implementation 
of NICs in preK—12 education, used the framework to evaluate implementation of the Min- 
nesota ALC NIC. The evaluation framework and its application to the Minnesota ALC NIC are 
described in this report. Use of the NIC evaluation framework can help other NICs in preK—12 
education determine whether they are implementing the essential features of NICs. 


The four essential features of a networked 
improvement community 


The processes followed by the Minnesota ALC NIC illustrate the four essential features that 
characterize NICs (Bryk et al., 2015). 


1. NICs are “focused on a well-specified common aim” (Bryk et al., 2015, p. 144) that is mea- 
surable and provides a shared focus. 


At the initial meeting of the Minnesota ALC NIC in 
September 2019, facilitators from the REL Midwest led Aim statement 
participants in a series of activities designed to specify 


a common aim. By the end of the meeting, participants By spring 2021 we will increase the 
had specified the following aim: By spring 2021 we will percentage of students who graduate with 
increase the percentage of students who graduate the necessary habits, skills, and knowledge 


with the necessary habits, skills, and knowledge to to achieve postsecondary success. 
achieve postsecondary success. 


2. NICs are “guided by a deep understanding of the problem, the system that produces 
the problem, and a theory of improvement relevant to it” (Bryk et al., 2015, p. 144). To 
develop this deep understanding, participants from different levels of the system engage 
in exercises to articulate the problem and its causes. For example, they might conduct a 
root-cause analysis to identify the many factors that contribute to the problem. 


At the September 2019 Minnesota ALC NIC meeting, administrators and teachers from the 
four ALCs that were then participants of the NIC engaged in a root-cause analysis and iden- 
tified a primary root cause: The school environment does not support students’ development 
of the habits, skills, and knowledge needed to graduate and achieve postsecondary success. 
They also identified several secondary root causes and then voted to focus on one second- 
ary root cause: Teachers are not setting clear expectations for student behavior or work. 


1. Three of the five participating ALCs were in suburban districts, and they were the only ALC in each of these 
districts. The remaining two ALCs were in urban districts. One of these urban ALCs was one of six in its dis- 
trict, and the other was one of nine in its district. Each participating ALC team was composed of administra- 
tors and teachers. 
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Drawing on the root-cause analysis, participants 


create a driver diagram that specifies hypotheses Cha nge idea: 

about the key factors—or drivers—that are likely to Student goal setting 

facilitate achieving their aim. 

Teachers and students work collaboratively 

3. NICs are “disciplined by the methods of improvement to set goals focused on gaining a deep 

research to develop, test, and refine interventions” understanding of concepts and skills. 

(Bryk et al., 2015, p. 144). Improvement research Students complete weekly “exit tickets” 

involves cycles of activities designed to test promis- to indicate whether they met their goals 

ing practices—also referred to as change ideas—for each day and reflect on their progress for 


addressing challenges (Improvement Science Research the week, including what they learned and 
Network, 2016). NIC participants use the technique what they could do differently in the future. 
known as Plan-Do-Study-Act cycles to test a change 
idea that applies the secondary driver. 


At the October 2019 meeting Minnesota ALC NIC participants developed a plan to test a 
change idea—in this case, a goal-setting activity in which teachers and students would 
meet to collaboratively set weekly goals. The plans included a timeline, participants’ roles 
and responsibilities, and steps for conducting the test. There were slight variations in the 
plans across ALC teams. In addition, the whole NIC group identified measures for monitor- 
ing the test of the change idea. 


4. NICs are “organized to accelerate the diffusion of these interventions out into the field 
and support their effective integration into varied educational contexts” (Bryk et al., 2015, 
p. 144). By testing similar interventions/change ideas in different contexts, NIC partici- 
pants learn what works, for whom, and in what context. They share their findings with 
one another and thus diffuse successful innovations into the field (Bryk et al., 2015). 


Figure 1. Timeline of the Plan-Do-Study-Act cycles of the Minnesota Alternative Learning 
Center Networked Improvement Community 


Note: The dashed line represents the COVID-19 pandemic period. The Minnesota Alternative Learning Center Networked 
Improvement Community could not meet because of the pandemic and associated school building closures. As a result, the 
third Plan-Do-Study-Act cycle, beginning in February 2020, was not completed. 


Source: Authors’ compilation. 
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At the December 2019 and February 2020 meetings, Minnesota ALC NIC participants 
shared their successes, challenges, and findings from data collection and analysis and 
decided how to adapt their change idea based on this information. 


The timeline for the Minnesota ALC NIC is in figure 1, and key continuous improvement 
activities—referred to as continuous improvement milestones—are in box 2. 


Box 2. Continuous improvement milestones 


Participants in networked improvement communities (NICs) engage in several key activities during the 
continuous improvement research process: 


Identifying a problem of practice. Participants identify a problem of practice that will be the focus of 
improvement efforts. 


Conducting a root-cause analysis. Participants identify root causes of the problem of practice. Often, 
participants use a fishbone diagram to create a visual display of the root causes of the problem. They 
insert a statement of the problem at the head of the fishbone diagram. Each major “bone” represents 
hypothesized root causes of the problem. Through discussions, participants further hypothesize about 
contributors to the root causes. 


Developing a driver diagram. Participants develop a visual display of NIC participants’ theory of the 
factors that likely contribute to achieving their aim. Primary drivers contribute directly to achieving the 
aim, while secondary drivers are components of primary drivers (Institute for Healthcare Improvement, 
2020). 


Identifying a change idea. Participants identify a practice to be designed, implemented, tested, and 
redesigned during a Plan-Do-Study-Act cycle. A change idea should target the root causes of the problem. 


Deciding how to measure implementation. NIC participants develop a plan for measuring and evaluat- 
ing whether the change idea is successful. 


Using findings to make decisions about Developing an implementation plan. 
next steps. NIC participants make NIC participants develop detailed 
decisions about next steps based on implementation plans that specify 
their analysis of the data. They roles, responsibilities, and timelines 
may choose to adopt a change for carrying out the implementation 


idea, adapt it, or abandon it. activities. 


Interpreting findings from data. 
NIC participants meet to review data 
collected during implementation. Often, 


Implementing the change idea. NIC 
participants implement the change idea. 
Collecting data. NIC participants collect 


they create data displays to more easily data to measure how well the program 
interpret the findings from the data is meeting its aim. 
collected. 


EVALUATION TOOLS AND FINDINGS 


REL Midwest researchers developed an approach to conducting a formative NIC evaluation 
and applied it to the Minnesota ALC NIC. The approach and the tools developed for the Min- 
nesota ALC NIC evaluation are described below. The tools can be adapted to evaluate other 
NICs in preK-12 education. The report also shares the findings from the Minnesota ALC NIC 
evaluation to illustrate how to analyze and interpret the data collected using these tools. 


Formative evaluation questions 


The purpose of a formative evaluation of an NIC is to answer questions about implementa- 
tion of the NIC (questions 1—5) and its progress toward achieving its aim (question 6). The 
evaluation tools presented in this report address these six research questions: 


1. To what extent are NIC participants engaged with NIC activities? How does this change 
over time? 


2. To what extent and in what respects do NIC participants perceive NIC activities to be rel- 
evant and useful? Which aspects of NIC meetings did participants find the most or least 
helpful and why? 


3. To what extent and in what respects are NIC participants gaining knowledge and skills to 
engage in continuous improvement? 


4. To what extent and in what respects are NIC participants completing continuous improve- 
ment milestones? How does this change over time? What challenges are associated with 
completing Plan-Do-Study-Act cycles? 


5. To what extent and in what respects are NIC participants coordinating their improvement 
efforts? How does this change over time? What challenges are associated with coordinat- 
ing improvement efforts? 


6. To what extent did the NIC make progress toward its aim? 


Each of these questions is discussed below in a separate section. The evaluation tools used 
to address each question are listed in table 1. A checklist for carrying out the evaluation 
using each tool is in appendix A. 


Engagement with networked improvement 
community activities 


Engaging a wide range of participants is critical to the success of an NIC in generating and 
testing useful continuous improvement ideas (Engelbart, 1992). Two indicators of engage- 
ment are discussed here: attendance at planned meetings and contributions to improve- 
ment milestones. 


Evaluation tools and findings 


Table 1. Application of evaluation tools to research questions for evaluation of the Minnesota Alternative 
Learning Center Networked Improvement Community 


Research 


VEY VEL ateyamceye)| question 


Tool description 


Attendance logs 1 Logs or some other means are used to track attendance at networked improvement community (NIC) 
meetings. 
Minnesota Alternative Learning Center Networked Improvement Community (Minnesota ALC NIC) 


facilitators created an attendance log and tracked attendance at each NIC meeting. 


Postmeeting surveys P28} Postmeeting surveys track participants’ perceptions of the usefulness and relevance of NIC meetings, 
as well as their knowledge and skills in engaging in continuous improvement. 

Minnesota ALC NIC facilitators administered surveys after whole-group NIC meetings. The evaluation 
team summarized the findings in memos to provide formative feedback to NIC facilitators (see 


appendix B for the postmeeting survey and appendix C for a sample memo). 


Post—Plan-Do-Study-Act 1,4,5 The post—Plan-Do-Study-Act survey measures participation in completing continuous improvement 
survey milestones and the extent to which participants interacted with staff from other ALCs. The survey 
also includes open-ended items about the challenges of completing continuous improvement 
milestones and coordinating improvement efforts. 
The evaluation team for the Minnesota ALC NIC administered the survey in February 2020 following 
the second Plan-Do-Study-Act cycle (see appendix D for the Post—Plan-Do-Study-Act cycle survey). 


Continuous improvement 4 
artifacts 


Continuous improvement artifacts are the work products created by NIC participants as they engage 
in activities to complete continuous improvement milestones (see box 2). These artifacts include 
meeting notes, fishbone and driver diagrams, implementation plans for Plan-Do-Study-Act cycles, 
data, and analyses. 

Two members of the evaluation team for the Minnesota ALC NIC examined these artifacts using a 
set of indicators to measure whether each milestone was completed (see table 3 later in the report). 
Data on student attendance, summarized in a data summary sheet, provided a leading indicator of 
progress toward the aim. (See exhibit E2 in appendix E for a data summary sheet template). 


Event summaries 5 Event summaries provide additional context for NIC evaluators, filling gaps in understanding when 
evaluation tools are inconclusive. 

Minnesota ALC NIC facilitators used a meeting agenda as a template for the event summaries and 
provided detailed notes on what was accomplished at each meeting. The evaluation team used 
these notes to clarify how the Minnesota ALC NIC fostered coordination of improvement milestone 


completion among different ALC teams. 


Source: Authors’ compilation. 


Attendance at planned meetings 


For the Minnesota ALC NIC the evaluation team tracked 
participants’ attendance at whole-group meetings. Min- 
nesota ALC NIC facilitators asked each ALC to list the staff 
who were participating in the NIC, and the facilitators 
then tracked attendance at each meeting, to determine 
whether participants were attending as planned. 


DY} e- ere) | (=rel nfo) amale) 


Provide an orientation to the goals and 
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The evaluation found strong engagement among teachers 
and administrators in the Minnesota ALC NIC, as mea- 
sured by attendance over time (table 2). All four partici- 
pating teachers and administrators from ALC A and ALC 

D attended all four meetings. ALC B sent a single admin- 
istrator to the first meeting, in September 2019, but the 
administrator then invited 19 teachers to attend future 
meetings, which they did. Three participating teachers 
and administrators from ALC C attended the September 


ivalem ol01n oke\t-Mo) mel lai-la-lalemer-lu- mere) |(-lelulo)a 
efforts, emphasize that the feedback 

from the evaluation is for improving NIC 
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the evaluation. 


Evaluation tools and findings 


Table 2. Each alternative learning center had 100 percent attendance of its participants at meetings of the 
Minnesota Alternative Learning Center Networked Improvement Community 


Number of participants attending meetin 
Alternative learning Total number of A ela AA A A P 8 8 


center (ALC) participants September 2019 (ofeh xo) ol-1ar40} |] December 2019 February 2020 


na is not applicable because the ALC was not participating at the time of the meeting. 


a. ALC B sent one administrator to the September 2019 meeting, who then invited 19 teachers to future meetings. 

b. ALC C left the NIC after the October 2019 meeting. 

c. ALC E joined in December 2019. Facilitators held separate meetings in December 2019 and January 2020 to bring ALC E participants up to speed, and 
then ALC E joined the whole-group meetings in February 2020. 


Source: Authors’ analysis of meeting attendance data. 


and October meetings, but they subsequently left the NIC because of competing demands at 
the ALC. ALC E joined the NIC in December 2019. The facilitators held separate meetings in 
December 2019 and January 2020 to bring ALC E participants up to speed, and all five teach- 
ers and administrators from the ALC joined the whole-group meetings in February 2020. 


Contribution to continuous improvement milestones 


A second indicator of engagement is the extent to which NIC participants contribute to con- 
tinuous improvement milestones (see box 2). Because some continuous improvement work 
occurs outside of meetings, attendance may not be an adequate indicator of contributions 
to milestones. To capture these external contributions, the evaluation team surveyed Min- 
nesota ALC NIC participants about their engagement in the milestones shortly after the con- 
clusion of each Plan-Do-Study-Act cycle (see question 2 in appendix D).* Respondents were 
asked to rate their level of participation in a particular continuous improvement milestone 
(for example, identifying a change idea that will influence drivers of improvement) using a 
four-point scale ranging from “not at all” to “very much so.” 


Minnesota ALC NIC facilitators administered the survey at the end of the February 2020 
meeting, which marked the conclusion of the second Plan-Do-Study-Act cycle. A majority 
of survey respondents felt that they had participated “moderately” or “very much so” in all 
the milestones except for two: developing a driver diagram and identifying a change idea 
(figure 2). This is possibly because these two milestones were carried out during the Sep- 
tember 2019 meeting, when only one administrator from ALC B attended and before the 19 
teachers from that ALC joined the NIC. 


The variation in the percentages of participants who felt that they had participated “moder- 
ately” or “very much so” provides formative feedback about the milestones that could use 


2. The post—Plan-Do-Study-Act survey can be administered in conjunction with the postmeeting survey rather 
than as a separate survey. 


Evaluation tools and findings 


Figure 2. A majority of survey respondents reported that they participated “moderately” 
or “very much so” in all continuous improvement milestones except developing a driver 
diagram and identifying a change idea 


Identifying a problem of practice 
Conducting a root-cause analysis 
Developing a driver diagram 

Identifying a change idea 

Deciding how to measure implementation 
Developing an implementation plan 
Implementing the change idea 

Collecting data 

Interpreting findings from data 


Using findings to make decisions about next steps 


fe) 20 40 60 80 100 


Percent reporting that they participated 
“moderately” or “very much so” in the milestone 


Note: The sample included 27 respondents to the post—Plan-Do-Study-Act survey. The response rate was 82 percent. 


Source: Authors’ analysis of data from a post—Plan-Do-Study-Act survey administered in February 2020. 


stronger support. For example, facilitators and participants in the Minnesota ALC NIC should 
consider why reported participation in the first four milestones in figure 2 varied so widely, 
from 31 percent to 61 percent, despite the fact that the NIC addressed all four of them 
during the September 2019 meeting. 


These findings are based on retrospective ratings that relied on survey respondents’ recall 

of their participation during the second Plan-Do-Study-Act cycle, which began two months 
prior to the survey. To limit recall bias, facilitators should include a field to record contribu- 
tor names on all NIC artifacts, which would allow NIC evaluators to determine participation 
in each continuous improvement milestone at the time of its completion. 


Relevance and usefulness of the networked 
improvement community 


After each whole-group meeting Minnesota ALC NIC participants completed a postmeeting 
survey that asked attendees to rate two statements about the relevance and usefulness of 
the meetings. The evaluation team used the Institute 
of Education Sciences’ Stakeholder Feedback Survey, 


created for participants in REL trainings, coaching ses- Analysis inf) 

sions, and events to provide feedback for use in improv- 

ing programs (see appendix B). Evaluators can assess Convert data from postmeeting surveys 
responses to the postmeeting survey and compare them into figures to provide timely feedback on 
over time to assess changes in the relevance and use- participants’ perceptions of NIC meetings. 
fulness of meetings and to examine variation across NIC Displaying findings from each session 
teams. side by side is useful in determining how 


perceptions change over time. 


Evaluation tools and findings 


The postmeeting survey was administered in paper and pencil format at the end of the Septem- 
ber, October, and December 2019 meetings. Response rates for meeting attendees were 100 
percent in the September meeting, 55 percent in the October meeting, and 43 percent in the 
December meeting. Administering surveys at a meeting can result in higher response rates and 
reduce recall bias. In this case the evaluation team hypothesized that the response rate declined 
in October and December because those meetings were held after school (rather than during 
school, as in September), and many participants had to leave immediately after the meeting.? 


Minnesota ALC NIC participants rated their agreement with the following statements about 
relevance and usefulness: 


e The coaching/consultation offered by the REL was relevant to a particular issue facing my 
agency or organization. 


e The benefits from this project were worth the time and resources my agency or organiza- 
tion invested to participate. 


Nearly all NIC participants who responded to the survey found the meetings to be relevant 
and useful to their organization (figure 3). Across all three meetings at least 90 percent of 
respondents “agreed” or “strongly agreed” that the coaching was relevant to a particular 
issue facing their organization. And at least 82 percent of respondents “agreed” or “strongly 
agreed” that the benefits were worth the time and resources that their agency or organiza- 
tion invested to participate. 


Figure 3. Most meeting attendees in the Minnesota Alternative Learning Center 
Networked Improvement Community agreed or strongly agreed that the meetings were 
relevant and useful to their organization 


m September 2019 m October 2019 m December 2019 


Relevant to my organization 


Worth the time and resources invested 


0 20 40 60 80 100 
Percent responding “agree” or “strongly agree” 
Note: The sample included 12 respondents in both September and December and 17 respondents in October. The response 


rate was 100 percent for September, 55 percent for October, and 43 percent for December. Participants rated items using a 
four-point agree/disagree scale. 


Source: Authors’ analysis of data from postmeeting surveys administered in September 2019, October 2019, and December 2019. 


3. The NIC facilitators suggested this explanation to the authors in a personal communication. 
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The postmeeting survey also included two open-ended items that addressed the useful- 
ness of the NIC by asking respondents to describe which aspects of the meeting were most 
helpful and which were least helpful. (A third open-ended item asked for suggestions for 
improving the meeting.) The evaluation team analyzed the responses using an inductive 
approach that assigned each comment to a category (for example, opportunities for team- 
work, strong facilitation). The team totaled the comments in each category to quantify the 
prevalence of each theme. 


When participants were asked what aspects of the meetings were most helpful and why, 
their comments clustered into three broad themes across the three meetings: 


e Team discussions. In October and December 2019 respondents most frequently identified 
ALC team discussions and opportunities to collaborate with other colleagues as most 
helpful. For example, in December, one respondent identified “discussion in small groups 
to process ideas” as the most helpful aspect of the meeting. 


e Development of relationships. In September 2019 three respondents noted the support- 
iveness of the coaches and shared their appreciation for the opportunity to work with 
them. Three respondents also reported that encountering participants of other ALCs was 
useful, while one respondent appreciated the opportunity to connect with peers in the 
respondent’s ALC. 


e Facilitation of continuous improvement. For all three meetings respondents mentioned 
facilitation as a helpful aspect. In September 2019 five respondents shared that they found 
specific techniques used by the facilitators to be helpful. One respondent reported, “I love 
how they broke items down and then built them back up so we get the full understanding 
of how to improve our schools.” In October two respondents mentioned the importance of 
modeling. One respondent stated, “Modeling the process and guiding us through the plan 
was very helpful and good models for students.” In December four respondents mentioned 
that the facilitators helped them reflect on what was working and what needed to change. 


Fewer respondents responded to questions about aspects of the coaching that were least 
helpful and how the coaching could be improved for the future. These responses were more 
diverse and did not cluster into themes across multiple respondents. 


Overall, the responses indicated that respondents viewed ALC team discussions and 
opportunities to develop relationships as critical to the usefulness of the Minnesota ALC 
NIC. These themes are consistent with the purpose of NICs, which is to foster a connected 
approach to improvement among individuals working within a system. 


Knowledge and skills for engaging in continuous 
improvement 


The evaluation team used self-reported data from the postmeeting surveys to assess Min- 
nesota ALC NIC participants’ knowledge and skills for engaging in continuous improvement. 
The postmeeting surveys asked NIC participants to rate their agreement with statements 
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about whether the coaching session increased their 


understanding of and skill in specific aspects of contin- Data collection ial) 
uous improvement covered during each session. For 
example, after the September 2019 meeting the survey To ensure that items are comparable 
asked participants about their knowledge and skills in across meetings, use generic terms and 
conducting a root-cause analysis and developing a driver refer to “the topic” or “this issue.” It is 
diagram. helpful to specify the topic addressed 

by the session in the introduction to the 
Beginning with the item stem, “The coaching/consultation survey to focus respondents’ attention on 
offered by the REL,” the survey included four item choices: the same topic when they are rating items. 


e Increased my understanding of the topic. 
e Increased my understanding of the research and data available to examine this issue. 


e Increased my understanding of the ways research and data can be used to investigate this 
issue. 


e Increased my capacity to use research and data to solve problems in my agency or 
organization. 


Nearly all respondents “agreed” or “strongly agreed” that participating in the September, 
October, and December 2019 meetings increased their understanding of the topic (figure 
4). For all other items most respondents “agreed” or “strongly agreed” that participating 

in the September meeting increased their knowledge and skills, but a higher percentage of 
respondents “agreed” or “strongly agreed” that participating in the October and December 
meetings increased their knowledge and skills: 


e More than 90 percent of respondents who participated in the October and December 
meetings “agreed” or “strongly agreed” that participating in the meetings increased their 
understanding of the research and data available to examine the issue compared with 73 
percent of respondents who participated in the September meeting. 


e More than 80 percent of respondents who participated in the October and December 
meetings “agreed” or “strongly agreed” that participating in the meeting increased their 
understanding of the ways research and data can be used to investigate the issue com- 
pared with 58 percent of respondents who participated in the September meeting. 


e More than 80 percent of respondents who participated in the October and December 
meetings “agreed” or “strongly agreed” that participating in the meeting increased their 
capacity to use research or data to solve problems compared with 58 percent of respon- 
dents who participated in the September meeting. 


One possible reason for the lower levels of agreement in September is that the questions refer 
to the entire coaching opportunity rather than to a specific meeting. Therefore, respondents 
at the October and December meetings were reflecting on the cumulative experience of the 
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Figure 4. In October and December 2019 more than 80 percent of meeting participants 
responded that the networked improvement community meeting increased their 
knowledge and skills 


m September 2019 m October 2019 m December 2019 


Increased my understanding of the topic 


Increased my understanding of the research 
and data available to examine this issue 


Increased my understanding of the ways research 
and data can be used to investigate this issue 


Increased my capacity to use research and data 
to solve problems in my agency or organization 


fe) 20 40 60 80 100 
Percent responding “agree” or “strongly agree” 
Note: The sample included 12 respondents in both September and December and 17 in October. The response rate was 100 
percent for September, 55 percent for October, and 43 percent for December. 


Source: Authors’ analysis of data from postmeeting surveys administered in September 2019, October 2019, and December 
2019. 


Minnesota ALC NIC rather than a single session. It is also possible that because the September 
meeting focused on the earlier milestones in the continuous improvement process, NIC partici- 
pants had not yet gained experience with working on milestones related to research and data. 


Completing continuous improvement milestones 
as intended 


The continuous improvement process generates artifacts that can be used to assess whether 
NIC participants are completing continuous improvement 
milestones, thus indicating that they are making progress 


toward their aim. Examples of artifacts are implementa- Analysis ia) 

tion plans, data collection forms, and data summaries. 

However, these artifacts are not always in a format that It is useful to present findings ina 

lends itself to analysis. To assist the analysis, evaluators crosswalk of teams by milestone (table 4). 
can coordinate with NIC facilitators to ensure that arti- The unit of analysis is a team of NIC 

facts align to continuous improvement milestones (see participants working together to complete 
box 2). For example, a completed driver diagram can a Plan-Do-Study-Act cycle. Sometimes 

be used as an artifact for developing a driver diagram an NIC has multiple teams (for example, 
milestone. Key indicators for this milestone are that the teams of teachers within an ALC). In the 
driver diagram has a measurable aim and specifies at Minnesota ALC NIC, three ALCs had a 
least one driver of the aim. For the Minnesota ALC NIC single schoolwide team, and one ALC had 
each continuous improvement milestone, the corre- three teams, one for each department. 


sponding artifact, and the indicators of milestone com- 
pletion are displayed in table 3. 
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Table 3. Continuous improvement milestones, artifacts, and indicators, with examples from the Minnesota 
Alternative Learning Center Networked Improvement Community (Minnesota ALC NIC) 


Milestone 


Identifying a problem 
of practice 


Conducting a root- 
cause analysis 


Developing a driver 
diagram 


Identifying a change 
idea 


Deciding how 
to measure 
implementation 


Developing an 
implementation plan 


Collecting data 


Interpreting findings 
from data 


Using findings to 
make decisions about 
next steps 


Artifact 


Meeting notes 


Fishbone diagram 


Driver diagram 


Meeting notes, 
presentation deck 


Meeting notes, 
presentation deck 


Implementation plan 


Data summary 


Data summary 


Data summary 


Level? 


NIC 


NIC 


NIC 


NIC 


NIC 


ALC 
team 


ALC 
team 


ALC 
team 


ALC 
team 


Indicator of completion/Minnesota ALC NIC example 


Identification of one problem of practice as the focus of the NIC. 
Example: “Many youth are not successfully developing the habits, skills, and knowledge 
needed to graduate and achieve postsecondary success.” 


Identification of at least one root cause. 
Example: School environment; student attitudes toward school. 


Agreement on a specific and measurable aim. 

Identification of at least one driver of the aim. 

Examples: 

¢ Specific and measurable aim: By spring 2021 we will increase the percentage of 
students who graduate with the necessary habits, skills, and knowledge to achieve 
postsecondary success by [specified] percent. 

¢ Driver of the aim: Improved school environment (primary driver); teachers set clear 
expectations for behavior and work (secondary driver). 


A description of a specific change idea that is related to a driver of the aim. 
Example: Teachers will work with each student to set goals and will support students in 
monitoring their goals. 


Specification of a measure of the driver to quantify whether the change idea has 
influenced the driver. 
Example: Percentage of students who are setting and achieving goals. 


Identification of individuals who will participate in implementation. 

Establishment of a timeline for implementation. 

Identification of predicted outcomes. 

Examples: 

e Individuals who will participate in implementation: Five students not on track to earn 
credit. 
Timeline: Students do goal check-ins for eight days. 

e Predicted outcomes: 80 percent of participating students will set a goal, 50 percent will 
accomplish goal. 


Compilation of data for analysis. 
Example: Teams compile number of students setting and accomplishing a goal. 


Summary of data that include a test of the predictions. 
Example: Teams compare predicted and actual goal setting and accomplishment by students. 


A decision about next steps (adopt, adapt, or abandon the change idea). 
Example: Turn paper goal sheet into a digital form. 


NIC is networked improvement community; ALC is alternative learning center. 


a. Level indicates whether the NIC completed the milestone as a whole group or separately within each ALC. 


Source: Authors’ compilation. 


The evaluation team for the Minnesota ALC NIC summarized the completion of milestones 
for the six ALCs participating in the December 2019 meeting.’ The evaluation team focused 
on the second Plan-Do-Study-Act cycle (December 2019—February 2020) because the facil- 
itators had used the first cycle for training. Ideally, evaluators would examine artifact data 
over multiple cycles, but subsequent Plan-Do-Study-Act cycles for the Minnesota ALC NIC 
were canceled because schools closed their buildings in March 2020 in response to the 
COVID-19 pandemic. The meeting notes indicated that the NIC worked on the first five mile- 
stones in table 3 as a group and the next four milestones as individual ALCs. 


4. ALCs A, D, and E had one team each. ALC B divided into three teams based on department. ALC C had left the 
NIC before the start of the second Plan-Do-Study-Act cycle. 
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The artifacts and indicators described in table 3 showed that the NIC—as a whole group— 
completed four of the five initial milestones. The meeting notes indicated that the Minne- 
sota ALC NIC had identified a problem of practice, but the milestone of developing a driver 
diagram was only partially completed. The aim statement, “By spring 2021 we will increase 
the percentage of students who graduate with the necessary habits, skills and knowledge 
to achieve postsecondary success by___—%,” included a placeholder that was never filled in 
with a specific percentage. Without specifying the percentage, the aim was not measurable 
and thus could not be used as an indicator of completion. 


To determine the extent to which individual ALC teams completed the final four milestones in 
table 3, the evaluation team examined two main artifacts (see appendix E for artifact templates): 


e An implementation plan written on chart paper by each ALC team working on a Plan-Do- 
Study-Act cycle. The format of the implementation plan was consistent across teams and 
included sections describing which students would participate in the change idea (setting 
and monitoring goals), when and how many times the team would implement the change 
idea, predictions for the outcomes of the change idea, preparation steps, and a descrip- 
tion of the steps involved in implementing the change idea. 


e Adata summary sheet with fields corresponding to the number of students setting and 
accomplishing goals, frequency of teacher/student reflections on goals, and student 
attendance. The data summary sheet also included fields for interpreting the data (such 
as whether predictions were accurate) and for considering next steps. 


Five of the six ALC teams completed the data collection and interpretation of findings mile- 
stones (table 4). Four of the six ALC teams developed an implementation plan and used find- 
ings to make decisions about next steps. 


Table 4. Most participating teams in the Minnesota Alternative Learning Center 
Networked Improvement Community completed all continuous improvement milestones 


Continuous improvement milestone completed 


Alternative Using findings to 
learning Developing an Interpreting findings make decisions 
center team? implementation plan Collecting data from data about next steps 
2 Nes Yes Yes No 

3 Yes Yes Yes Yes 

4 Yes Yes Yes Yes 

5 No Yes Yes Yes 

6 No No No No 

Total “yes” 4 5 5 4 


a. In the Minnesota Alternative Learning Center Networked Improvement Community, alternative learning centers (ALCs) A, 
D, and E had one team each; ALC B had three teams based on department; and ALC C left before the start of the second Plan- 
Do-Study-Act cycle, the focus of the evaluation. 


Source: Authors’ analysis of data from Plan-Do-Study-Act cycle artifacts produced between December 2019 and February 
2020. 


aks) 


Evaluation tools and findings 


The February 2020 post—Plan-Do-Study-Act survey 
included an open-ended item asking respondents to 
describe the challenges they or their ALC team encoun- 
tered in completing the Plan-Do-Study-Act cycle. 
Respondents listed several challenges, all of which 
related to implementation of the change idea rather than 
to the continuous improvement process for studying the 
change idea. Teachers noted low student attendance 

(12 teachers), difficulty explaining goal setting to students 
(6 teachers), and trouble finding the time to set goals (6 
teachers) as problems encountered in implementing goal 
setting. 


Coordinating continuous 
improvement efforts 


Analysis tip 


To analyze data on challenges in completing 
milestones, create a set of codes, each 
corresponding to a type of challenge (for 
r)'€Jan]o)(=PReol¥] (0M alo) maelen])(=1t Me al-lal xem (el-t-) 
data analysis was too time-consuming, 
(xo) (eM alolmi-¥-4n2\-Mo) alr-M 0) a=1el(einlo)a) Mie aalaaliatsd 
the number of respondents reporting each 
challenge will help NIC facilitators prioritize 
WV altel alelat-)i(=lal-xemnomr-lelel gation 


An essential feature of NICs is that their participants coordinate their efforts to accelerate 
the design, implementation, testing, and redesign of change ideas. By testing similar change 
ideas in different contexts, NIC participants learn what works, for whom, and in what 
context. The NIC evaluation team used the February 2020 post—Plan-Do-Study-Act survey 
to assess the coordination of improvement efforts in the Minnesota ALC NIC. Respondents 
rated their agreement with six statements about their interactions with other ALCs. The 
statements were related to sharing Plan-Do-Study-Act cycle findings (two statements), 
discussing improvements efforts, sharing resources related to improvement, learning from 
other ALCs’ findings, and benefiting from discussions with other ALCs (figure 5). 


Figure 5. Fewer than half of survey respondents reported coordinating continuous 
improvement efforts with other alternative learning center teams in the Minnesota 
Alternative Learning Center Networked Improvement Community 


The other ALCs in this NIC shared the 
findings from their PDSA cycle(s) 


Our ALC shared findings from our PDSA cycle(s) 
with staff from the other ALCs in this NIC 


Our ALC has shared resources related to 
improvement with the other ALCs in this NIC 


Our ALC learned from the findings 
of the other ALCs in this NIC 


Our ALC has benefited from discussions 
with the other ALCs in this NIC 


Our ALC had discussions about our improvement 
efforts with staff from the other ALCs in this NIC 


0) 25 


75 100 


Percent agreeing “moderately” or “very much so” with statement 


NIC is networked improvement community; PDSA is Plan-Do-Study-Act cycle. 


Note: The sample included 26 respondents to the February 2020 post—Plan-Do-Study-Act survey. The response rate was 79 


percent. 


Source: Authors’ analysis of data from a post—Plan-Do-Study-Act survey administered in February 2020. 
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Minnesota ALC NIC participants’ responses to the post—Plan-Do-Study-Act survey indicated 
that networking among the ALC teams occurred among a minority of participants during 
the second Plan-Do-Study-Act cycle. Fewer than one-third of the 26 survey respondents 
indicated that they “moderately” or “very much so” shared resources related to continu- 
ous improvement efforts with the other ALC teams in the NIC (32 percent) or learned from 
the findings of the other ALC teams in the NIC (28 percent; see figure 5). Fewer than half 

of respondents indicated that other ALC teams in the NIC “moderately” or “very much so” 
shared findings from their Plan-Do-Study-Act cycles (42 percent) or that their ALC team 
“moderately” or “very much so” shared findings from their Plan-Do-Study-Act cycles with 
other ALC teams in the NIC (44 percent), had discussions about their improvement efforts 
with other ALC teams (44 percent), or benefited from discussions with other ALC teams (48 
percent). 


The post—Plan-Do-Study-Act survey included an open-ended item asking respondents about 
the challenges their ALC team encountered when coordinating efforts with the other ALC 
teams participating in the NIC. Few respondents, however, described specific challenges, and 
five respondents reported that they did not work with other ALC teams. 


Written summaries of NIC meetings can provide additional context to fill in gaps in under- 
standing arising from nonresponses. For example, notes added to the event summary com- 
pleted by the facilitator after the February 2020 meeting provided additional insight into the 
extent to which Minnesota ALC NIC participants were coordinating continuous improvement 
efforts: 


Facilitators circulated and helped groups identify changes to their approach, 
always reflecting on what the data they gathered revealed. [ALC E] has only 
completed one cycle of implementation. The facilitators spent time helping 
them review their data and make adjustments to their implementation plan. 
[They] identified that they need reminders to implement and would like to 
move to a longer trial period to catch students who are chronically absent. 
They are targeting students who were not regularly connecting with adults 

in the school. They will also be using the GOALS form created by [ALC A] to 
increase student reflection in the activity. [ALC A] decided to add the goal of 
getting students to set mastery goals this next cycle. Several of the [ALC B] 
teams discussed how they would add “fun” elements to the goal-setting activ- 
ity, such as celebrations for meeting a certain number of goals and improving 
the environment in which they set goals with students. [ALC D] decided to 
adopt the modified goals from [ALC B] and to lengthen the number of days in 
each trial cycle to catch students who have attendance problems. 


This event summary shows that as ALC teams progressed through Plan-Do-Study-Act cycles, 
they modified their change idea based on the experiences of other teams. This finding is 
evidence of coordinating continuous improvement efforts. It also illustrates the importance 
of supplementing survey data with other sources of information about coordinating efforts. 
At the time of survey completion, NIC participants might be unable to recall these exchanges 
or might not realize that they are examples of coordinating continuous improvement efforts. 
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Another method for examining the extent of networking is for NIC teams to record each 
change idea that they test for the duration of the NIC and document the source for the 
change idea. Teams could log their reason for choosing the change idea and note whether 
they got the idea from another team. This log will allow the NIC team and the evaluation 
team to observe how ideas change in response to testing. 


Although social network analysis was not used in this evaluation, other evaluators have used 
it to map connections among different NIC teams (Feldstein & Sherer, 2018). Existing data, 
such as data from an online community of practice, can be used to capture the nature and 
frequency of communications among NIC participants.° In the absence of such data, surveys 
can inquire about the extent of connections among NIC teams. Software packages are avail- 
able to generate maps of network connections that identify central actors in the NIC and 
highlight any potentially problematic isolation of NIC teams (Feldstein & Sherer, 2018). Social 
network analysis, however, requires specialized skills that make it more challenging to use 
than the tools presented in this report. 


Progress toward the aim 


While the preceding questions guiding the evaluation of the Minnesota ALC NIC focused 

on implementation of the NIC, the final evaluation question focuses on outcomes: Did the 
NIC make progress toward attaining its stated aim? One challenge in evaluating progress 

is that the aim of an NIC is often too distal to be of use for continuous improvement. Aims 
usually relate to policy-critical outcomes such as teacher retention, graduation rates, and 
performance on annual tests. Data on these outcomes are typically reported annually, which 
does not allow for multiple rounds of testing and feedback during a school year. Therefore, 
to evaluate an NIC’s progress toward its aim, it is helpful to measure progress relative to 

the identified drivers of the aim. The change idea exerts a direct influence on these drivers, 
which in turn influence the aim. 


The Minnesota ALC NIC expected increased student goal setting to lead to higher student 
attendance (as students are more engaged), to higher course completion rates, and ulti- 
mately to higher graduation rates. For this reason the NIC collected data on student atten- 
dance during the testing of the change idea. 


Prior to implementing the change idea, it is important to establish a baseline to serve as a 
comparison. Because the change idea in the Minnesota ALC NIC was implemented over a 
two-week period, teachers could calculate a baseline by determining attendance rates for 
one or more two-week periods preceding the introduction of the change idea. The more 
data points collected prior to and after the introduction of the change idea, the clearer 
patterns will be. A hypothetical example in figure 6 illustrates trends in attendance data 
collected by five teachers.® 


5. This type of analysis was not possible for the Minnesota ALC NIC evaluation because data on individual con- 
nections were not available. 

6. Because of school building closures related to the COVID-19 pandemic, teachers participating in the Minne- 
sota ALC NIC were not able to collect sufficient attendance data to evaluate progress toward the aim 
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This visualization of the simulated dataset reveals a trend toward increased attendance in 
the three weeks following the introduction of the change idea. Visualizing the data in this 
way is useful for low-stakes decisions about the progress of an NIC. When higher precision 
is required, quantitative approaches, such as an interrupted time series analyses, are more 
appropriate for conducting an analysis. 


Figure 6. Hypothetical data illustrating an average increase in student attendance 
following the introduction of the change idea 
= Teacher1 =— Teacher2 = Teacher3 == Teacher4 = Teacher5 *** Average 


Attendance rate (percent) 
80 


eoccece 


60 


40 


20 


Week -3 Week —2 Week -1 Week 1 Week 2 Week 3 


Source: Authors’ analysis of hypothetical data. 
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IMPLICATIONS AND LIMITATIONS 


The evaluation of implementation of the Minnesota ALC NIC has several implications and 
limitations: 


e Attendance data and surveys of member involvement with continuous improvement 
milestones were used to assess engagement in the Minnesota ALC NIC, which was found 
to be strong. Attendance data were easy to obtain, as NIC facilitators recorded the names 
of everyone who attended each session. Findings from the February 2020 post—Plan- 
Do-Study-Act survey indicated weaker engagement with some milestones than others. 

A limitation of the survey is that it relied on the recollections of NIC participants. In the 
future it would be useful for NIC participants to record participation in each continuous 
improvement milestone as they work on it. 


e Participants considered the Minnesota ALC NIC to be both relevant and useful, as indi- 
cated by their responses to postmeeting surveys. The most frequent reasons given were 
the opportunities to work in ALC teams and to develop relationships with staff from other 
ALCs. Also, participants perceived the quality of the facilitators to be critical to the use- 
fulness of the NIC. The low response rates to some of the postmeeting surveys, however, 
made it difficult to compare survey results across meetings. To overcome this impedi- 
ment, facilitators could add completing the survey to the meeting agenda and provide 
incentives for completing the survey. 


e The percentage of attendees agreeing that the NIC increased their knowledge and skills 
grew over time, although the low response rates on the later postmeeting surveys limited 
comparisons across meetings. Further, the survey questions about knowledge and skills 
referred to the coaching/consultation experience overall and not to a particular session. 
While this strategy is useful to understand how participants perceive the NIC process as a 
whole, it makes it is more difficult to compare perceptions across sessions. 


e Artifacts from the Minnesota ALC NIC enabled the evaluation team to conclude that 
most of the ALC teams completed most of the continuous improvement milestones. The 
ALC teams used a consistent format for their implementation plans and data summaries, 
which facilitated the review. But as complete data were available for only one of the three 
Plan-Do-Study-Act cycles, the evaluation team could not determine how ALC teams modi- 
fied their change idea over time in response to their findings. 


e While the February 2020 post—Plan-Do-Study-Act survey indicated that sharing was 
infrequent among ALC teams in the NIC, contemporaneous meeting notes recorded spe- 
cific instances of networking among teams. It is possible that the networking involved a 
limited number of participants. In the future, ALC teams could be asked to keep a log of 
their change ideas and record whether they adopted ideas from other teams, to facilitate 
evaluation. 
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The evaluation findings are limited in other respects as well: 


e The changing composition of the Minnesota ALC NIC affected the evaluation. An ALC that 
joined the NIC two months late required one-on-one workshops to catch up with the 
others. Because this ALC received a different level of support than the others received, 
the evaluation team analyzed two different versions of NIC supports at the same time. 


e The evaluation team had intended to administer two additional post—Plan-Do-Study- 
Act cycle surveys in spring 2020, but the work of the NIC was interrupted by the school 
building closures in response to the COVID-19 pandemic. This precluded any evaluation of 
changes over time in completion of milestones and coordination of continuous improve- 
ment efforts, among other areas. 


e Because of the limited data available, the evaluation team could not examine the progress 
of the Minnesota ALC NIC toward its aim or drivers. 


The tools described and presented in this report can be used and adapted as needed to 
evaluate implementation of a range of NICs in preK—12 education settings. It is important for 
NIC facilitators to consider ways to incorporate data collection into the NIC, such as through 
attendance records, surveys, and other artifacts. Evaluation activities should be an integral 
part of the overall planning of the NIC. The planning should specify the timing of data col- 
lection related to NIC events, which participants will support data collection, and how the 
NIC teams will document their continuous improvement activities in ways that can support 
analysis. The evaluation planning checklist (see appendix A) includes specific steps for coor- 
dinating the evaluation with overall NIC facilitation. 
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APPENDIX A. EVALUATION 
PLANNING CHECKLIST 


Evaluators of networked improvement communities (NICs) can use this checklist for planning 
the evaluation. For most data sources an additional planning step is to determine a protocol 
for the secure transfer of data to the evaluation team. 


Attendance logs 

Create a list of NIC participants that includes alternative learning center (ALC) affilia- 
tions and contact information. 

Create an attendance tracking sheet for in-person meetings. 

Determine who will be responsible for tracking attendance. 


Postmeeting survey 

Obtain a schedule of meetings from NIC facilitators. 

Review goals of each NIC meeting and update the survey accordingly. 

Determine who will administer the survey and the mode of administration (paper and 
pencil or online). 

If the survey will be administered online, program it into the online survey platform. 
If the survey will be administered by paper and pencil, ensure that there are enough 
copies to distribute at the meeting. 


Post—Plan-Do-Study-Act survey 

Obtain a schedule of the Plan-Do-Study-Act cycles, and schedule the survey for the 
conclusion of each cycle. 

Determine who will administer the survey and the mode of administration (paper and 
pencil or online). 

If the survey will be administered online, program it into the online survey platform. 
If the survey will be administered by paper and pencil, ensure that there are enough 
copies to distribute at the meeting. 


Continuous improvement artifacts 

Get information about the team structure within ALCs. Are there multiple teams 
within each ALC? Who belongs to which team? Each team conducting the Plan-Do- 
Study-Act cycles should generate its own set of continuous improvement artifacts. 
Determine how the Plan-Do-Study-Act artifacts will be archived. Where will they be 
saved? Who will be responsible for archiving them? 

Communicate with NIC participants about expectations for completing the artifacts. 
Encourage teams to annotate artifacts in the following ways: 

Record which participants participated in the creation of each artifact. 

For the Plan-Do-Study-Act planning form, record the source for the change idea. 
Determine how the Plan-Do-Study-Act artifacts will be archived. Where will they be 
saved? Who will be responsible for archiving them? 


Event summaries 
Designate a notetaker for each meeting. 
Record notes in the agenda to annotate each component of the meeting. 


APPENDIX B. POSTMEETING SURVEY 


The items for the postmeeting survey were drawn from the Institute of Education Sciences’ 
Stakeholder Feedback Survey, a survey created to provide feedback on program improve- 
ment for Regional Educational Laboratory (REL) trainings, coaching sessions, and events. 
Using the postmeeting survey, evaluators can analyze data over time to assess changes in 
the relevance and usefulness of meetings and to examine variation among alternative learn- 
ing center (ALC) teams. 


Minnesota Alternative Learning Center 
Networked Improvement Community: 
Postmeeting survey 


1. For the questions below, please indicate the extent to which you agree or disagree with 
the following statements. 


Strongly Strongly 
disagree Disagree Agree agree 
a. The coaching/consultation offered by the REL was relevant to = — = Oo 
a particular issue facing my agency or organization 
b. The coaching/consultation offered by the REL increased my 
: : Oo O O 
understanding of the topic 
c. The coaching/consultation offered by the REL increased my 
understanding of the research and data available to examine L a [I Oo 
this issue 
d. The coaching/consultation offered by the REL increased my gO O gO oO 


awareness of available research and data on this topic or issue 


e. The coaching/consultation offered by the REL increased my 
understanding of the ways research and data can be used to Oo 
investigate this issue 


f. The coaching/consultation offered by the REL increased my 
capacity to use research and data to solve problems in my O i) SI O 
agency or organization 


g. | would likely participate in additional support activities 


offered by the REL O 

h. | understood my role and the expectations for my gO = g gO 
participation in this project ay 

i. My agency or organization received the appropriate level of g — g Oo 
support from the REL to implement this project _ 

j. The benefits from this project were worth the time and oO = g oO 


resources my agency or organization invested to participate 


2. What aspects of the coaching/consultation were most helpful and why? 


Appendix B. Postmeeting survey 


3. What aspects of the coaching/consultation were least helpful and why? 


4. What part of this coaching/consultation would you suggest changing to make it better for 
future participants? 
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APPENDIX C. PARTICIPANT 
FEEDBACK MEMO EXAMPLE 


On September 9 and 10, 2019, the Regional Educational Laboratory (REL) Midwest, in collab- 
oration with the Minnesota Department of Education, held a two-day coaching session for 
12 teachers and administrators from four Minnesota alternative learning centers (ALCs). The 
participants were from the Minnesota Alternative Learning Center Networked Improvement 
Community (Minnesota ALC NIC), a collaborative research partnership focused on improving 
the implementation of competency-based approaches to credit recovery in Minnesota ALCs 
through iterative cycles of design, implementation, and testing. Ultimately, the participants 
hope to increase graduation rates for ALC students. 


During the first day of coaching, the REL Midwest project team facilitated a root-cause 
analysis to identify specific, actionable challenges to implementing competency-based 
approaches to credit recovery and to determine the root causes of those challenges. Par- 
ticipants in the networked improvement community (NIC) came to a consensus about the 
challenge they would focus on during the first year of the NIC. That challenge was: “Many 
youth are not successfully developing the habits, skills, and knowledge needed to graduate 
and achieve postsecondary success.” 


During the second day of the session, REL Midwest coaches led participants in developing a 
theory of action related to the challenge. First, participants identified the following aim: “By 
spring 2021 we will increase the percentage of students who graduate with the necessary 
habits, skills, and knowledge to achieve postsecondary success.” Next, they identified the 
primary drivers that would help them achieve that aim and chose to focus on improving the 
school environment to support students’ development of the habits, skills, and knowledge 
needed to graduate and achieve postsecondary success. They identified four secondary 
drivers of this primary driver: improving the school’s physical environment, providing accom- 
modations or differentiated instruction, scaffolding learning opportunities, and setting clear 
expectations for student behavior and work. Participants voted to focus on setting clear 
expectations for students. 


The Minnesota Department of Education administered an anonymous survey to collect feed- 
back from NIC participants about their perceptions of the meeting’s usefulness and rele- 
vance. The Minnesota Department of Education returned the completed surveys for analysis 
by a separate group from REL Midwest, the REL Midwest evaluation team. This memo shares 
the findings from the evaluation team’s analysis of the survey data. 


Focus of feedback survey 


The feedback survey included 10 statements about the coaching session and asked par- 
ticipants to express their agreement with each (see appendix B for the full survey). The 
statements were related to the benefits, relevance, and usefulness of the coaching and to 
the support and expectations for participation in the coaching. Participants were also asked 
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open-ended questions about the most helpful and the least helpful aspects of the coaching 
session, as well as how the coaching session could be improved for future participants. 


Ratings of the coaching session 


All 12 participants in the coaching session completed the survey. Participants expressed 
strongest agreement with five statements related to relevance, increased knowledge of 
topic, expectations for participation, support for participation, and usefulness (expressed as 
“benefits of participation”): 


e The coaching/consultation offered by the REL was relevant to a particular issue facing my 
agency/organization. 


e The coaching/consultation offered by the REL increased my understanding of the topic. 
e |understood my role and the expectations for my participation in this project. 


e My agency or organization received the appropriate level of support from the REL to 
implement the project. 


e The benefits from this project were worth the time and resources my agency or organiza- 
tion invested to participate in the project. 


Participants expressed weaker agreement with three statements related to the coaching’s 
influence on their ability to use research and data: 


e The coaching/consultation offered by the REL increased my understanding of the research 
and data available to examine this issue. 


e The coaching/consultation offered by the REL increased my understanding of the ways 
research and data can be used to investigate this issue. 


e The coaching/consultation increased my capacity to use research and data to solve prob- 
lems in my agency or organization. 


Individual participants responded as follows: 

e Relevance. Eight participants “strongly agreed” and three participants “agreed” that 
the coaching was relevant to a particular issue facing their organization. One participant 
selected “not applicable.” 

e Usefulness. Eight participants “strongly agreed” and four participants “agreed” that the 


benefits from this project were worth the time and resources that their organization 
invested to participate. 
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e Knowledge and skills. Participants responded to five statements related to the coaching’s 
influence on their knowledge and skills: 


o Three participants “strongly agreed” and five participants “agreed” that the coaching 
offered by the REL increased their awareness of the research and data available on this 
issue. Four participants selected “not applicable.” 


o Six participants “strongly agreed” and six participants “agreed” that the coaching 
offered by the REL increased their understanding of the issue. 


° Two participants “strongly agreed” and six participants “agreed” that the coaching 
offered by the REL increased their understanding of the research and data available 
to examine this issue. Three participants selected “not applicable,” and one did not 
respond to this item. 


© Two participants “strongly agreed,” five participants “agreed,” and one participant 
“disagreed” that the coaching offered by the REL increased their understanding of the 
ways research and data can be used to investigate this issue. Four participants selected 
“not applicable.” 


o Three participants “strongly agreed,” four participants “agreed,” and three partici- 
pants “disagreed” that the coaching offered by the REL increased their capacity to use 
research and data to solve problems in their organization. Two participants selected 
“not applicable.” 


e Satisfaction. Five participants “strongly agreed,” six participants “agreed,” and one partic- 
ipant “disagreed” that they would likely participate in additional support activities offered 
by the REL. 


e Expectations for participation. Five participants “strongly agreed” and seven participants 
“agreed” that they understood their role and the expectations for their participation in 
the project. 


e Support for participation. Six participants “strongly agreed” and four participants 
“agreed” that their organization received the appropriate level of support from the REL to 
implement this project. Two participants selected “not applicable.” 


Open-ended responses 


e When asked what aspects of the coaching were most helpful and why, participants 
offered the following responses: 


© Developing relationships. Three participants noted the supportiveness of the NIC coaches 
and shared their appreciation for the opportunity to work with them. Three participants 
reported that the opportunity to meet participants from other ALCs was useful, and one 
participant shared appreciation for the opportunity to connect with peers in their ALC. 
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© Facilitation. Five participants found specific techniques used by coaches to be helpful. 
One participant reported, “I love how they broke items down and then built them back 
up so we get the full understanding of how to improve our schools.” 


e When asked what aspects of the coaching were least helpful and why, one participant 
noted that “complaining about broken issues” was not useful. Another participant wanted 
“more explanation ahead of time to know our purpose.” A third participant felt that the 
time of the meeting was not ideal and would have preferred a morning meeting. 


e When asked how coaching could be improved for future participants, four participants 
offered the following: 


© Session organization. One participant wanted to know the “big picture,” sharing that 
the support offerings were “too vague.” Similarly, another participant wanted “a visual 
layout to get a feel for how the sessions play out.” 


© Session logistics. One participant shared that the session would be better if it were 
held during the day and at the participant’s school. 


° Coaching quality. One participant requested that the coaches explain acronyms. 
Conclusions 


These findings suggest that participants had positive impressions of the coaching overall. 
There was room for improvement, however, concerning statements about the usefulness of 
the coaching on research and data. The open-ended comments will allow the NIC facilitators 
to build on a few strengths, including the facilitators’ supportiveness and the opportunities 
provided for interaction with teachers and administrators at other ALCs. Facilitators might 
consider providing a clear roadmap for future sessions. 
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APPENDIX D. POST—PLAN-DO- 
STUDY-ACT CYCLE SURVEY 


The post—Plan-Do-Study-Act survey is a brief survey administered shortly after the con- 
clusion of each Plan-Do-Study-Act cycle. Respondents use a four-point scale to rate their 
participation in continuous improvement milestones. To track change over time, the surveys 
should be administered after each cycle. 


The items in the survey can be tailored to specific networked improvement communities 
(NICs). The evaluation team for the Minnesota Alternative Learning Center Networked 
Improvement Community, for example, chose to use change idea rather than intervention 
because that was the language used by NIC facilitators. The evaluation team also embedded 
the specific data collection tool (exit tickets) in one of the survey items so that respondents 
could recognize it more readily. The survey includes items that ask respondents to rate the 
extent to which their alternative learning center (ALC) interacted with other ALCs in the NIC. 


Minnesota Alternative Learning Center 
Networked Improvement Community: Post—Plan- 
Do-Study-Act survey 


The purpose of this survey, administered by the Minnesota Department of Education, is 

to gather feedback from participants in the Minnesota Alternative Learning Center Net- 
worked Improvement Community (Minnesota ALC NIC). The survey asks about your par- 
ticipation in and perceptions of the Minnesota ALC NIC. Your responses will be used to 

help make the NIC more useful and relevant to the work you do in ALCs. The survey should 
take about 5-10 minutes to complete. Your responses will be analyzed by researchers 

from the Regional Educational Laboratory (REL) Midwest and will be completely confidential. 


1. What is the name of your ALC? Please circle your response. 
a. ALCA. 
b. ALC B. 
c. ALC D. 
d. ALC E. 


The following questions refer to the Plan-Do-Study-Act (PDSA) cycle that your ALC devel- 
oped as part of the Minnesota ALC NIC (starting in fall 2019). 


Appendix D. Post—Plan-Do-Study-Act Cycle survey 


2. Please indicate the extent to which you personally participated in each of the following 
tasks related to your ALC’s PDSA cycle. Please circle your response. 


Very 


Not at all Minimally Moderately allo sexe) Don’t know 


a. Identifying a “problem of practice” that 
will be the focus of the NIC’s improvement Oo = O 
efforts. 


io” 


. Conducting a “root-cause analysis” to 
describe the causes of the problem of O O I) O O 
practice. 


c. Developing a “driver diagram,” which 
is a visual representation of the drivers oq Oo oO 
(or factors) that are expected to lead to 
improvement. 


d. Identifying a specific “change idea” that 
pate 3 : O O O 
will influence drivers of improvement. 


e. Deciding how to measure implementation g Oo oO 
of the change idea. 


fe Developing plans for implementing the oO oa = = Oo 
change idea. 


g. Implementing the change idea in your Oo 
classroom. 


a 


. Collecting data using exit tickets (goal oO gO = = oO 
forms) and the PDSA trial tracker. 


. Interpreting findings from data to 
understanding if implementation of the O [ O 
change idea was successful. 


j. Using findings from data to make = — 
decisions about next steps. q q q 


3. Did you or your ALC colleagues encounter challenges with completing any of the PDSA 
activities described? Please describe these challenges. 


4. Please indicate the approximate percentage of staff from your ALC with whom you have 
interacted about this NIC. Please circle your response. 


a. None. 

b. 1-20 percent of staff. 

c. 21—40 percent of staff. 
d. 41—60 percent of staff. 
e. 61-80 percent of staff. 


f. 81-100 percent of staff. 
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5. Please indicate the extent to which your ALC interacted with the other ALCs in this NIC. 


a. The other ALCs in this NIC shared the 
findings from their PDSA cycle(s). 


io” 


. Our ALC shared findings from our PDSA 
cycle(s) with staff from the other ALCs in 
this NIC. 


c. Our ALC had discussions about our 
improvement efforts with staff from the 
other ALCs in this NIC. 


d. Our ALC has shared resources related to 
improvement with the other ALCs in this 
NIC. 


e. Our ALC learned from the findings of the 
other ALCs in this NIC. 


f. Our ALC has benefited from discussions 
with the other ALCs in this NIC. 


Not at all 


0 


i] 


Minimally 


Oo 


O 


Moderately 


Very 
alvClsexe) 


0 


O 


know 


6. What challenges, if any, did your ALC encounter when coordinating efforts with the other 


ALCs participating in the NIC? 


Thank you for participating in this survey! 
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APPENDIX E. ARTIFACT TEMPLATES 


The template in exhibit E1 can be used by networked improvement community participants 
to create a detailed plan for implementing the change idea/s. 


Exhibit E1. Implementation plan 


Alternative learning center name: 
Change idea tested: 
PDSA cycle no.: Description 


Where and with whom will this change idea be 
implemented? 


Who will be the primary person responsible for 
implementing this change idea? Which team participants 
will be supporting the implementation of this change idea? 


What are the key action steps for implementing this change 
idea? 


What is the timeline for implementing these action steps? 


Describe an early win for the implementation of this change 
idea. 


Describe one or two challenges to the implementation of 
this change idea. 


What supports or resources are essential for implementing 
this change idea? 


Source: Adapted with permission from Rowland el al. (2018). 


Appendix E. Artifact templates 


The template in exhibit E2 can be adapted to collect and compile data related to the change 
idea. This particular summary sheet was designed to collect data on a goal-setting inter- 
vention, including data on the number of students setting goals each week, the number of 
teacher/student reflections on those goals, and daily student attendance. The sheet can be 
adapted for other types of interventions and measures. Participants can enter their interpre- 
tations of the data and their planned next steps at the bottom of the form. 


Exhibit E2. Data summary sheet 


Change Idea Tested: Goal setting PDSA Cycle #: 1 


Total Number of Students: 15 


Date Measure Goal Observed 
Week 1 Number of students setting goals 12 10 
(exeinele) Number of teacher/student reflections 12 8 
Daily student attendance 14 M T w Th 
14 13 13 14 11 
Week 2 Number of students setting goals 


Number of teacher/student reflections 


Daily student attendance M T w Th F 


Week 3 Number of students setting goals 


Number of teacher/student reflections 


Daily student attendance M T w Th F 


Week 4 Number of students setting goals 


Number of teacher/student reflections 


Daily student attendance M T Ww Th F 


Reflection: 


Next Steps: 


Source: Authors’ compilation. 
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